-
-
Notifications
You must be signed in to change notification settings - Fork 177
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Napi rewrite #540
Napi rewrite #540
Conversation
@@ -37,8 +37,8 @@ test('test destroy non-existent directory', function (t) { | |||
rimraf(location, { glob: false }, function (err) { | |||
t.ifError(err, 'no error from rimraf()') | |||
|
|||
leveldown.destroy(location, function () { | |||
t.is(arguments.length, 0, 'no arguments returned on callback') | |||
leveldown.destroy(location, function (err) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This behavior has changed. The callback now return null
.
@@ -55,8 +55,8 @@ test('test destroy non-existent parent directory', function (t) { | |||
|
|||
t.notOk(fs.existsSync(parent), 'parent does not exist before') | |||
|
|||
leveldown.destroy(location, function () { | |||
t.is(arguments.length, 0, 'no arguments returned on callback') | |||
leveldown.destroy(location, function (err) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same here.
/** | ||
* Returns a context object for a database. | ||
*/ | ||
NAPI_METHOD(db_init) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've changed the model a bit. Either a NAPI_METHOD
returns a context, which all methods ending with _init
does, or a method takes a context. This is different from the previous implementation, where you had methods on objects, sort of like a js object if you will. This is more c style.
/** | ||
* Puts a key and a value to a database. | ||
*/ | ||
NAPI_METHOD(db_put) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
All NAPI_METHOD
's have their corresponding worker next to it. So it's easy to find.
} | ||
|
||
virtual void DoExecute () { | ||
SetStatus(database_->Get(options_, key_, value_)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here I mimic the behavior of the AsyncWorker
class in nan
. We store status and if status is ok, we call back will null
as a default action. In case a method wants to call back with other data, e.g. in NAPI_METHOD(db_get)
, its corresponding worker (GetWorker
) will override HandleOKCallback()
, see below.
/** | ||
* Worker class for deleting a value from a database. | ||
*/ | ||
struct DelWorker : public BaseWorker { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe you wonder why I'm using struct
and not class
here. A struct
in C++ is a class
. The only difference is default access. On a struct
everything is public
by default. On a class
it's private
. I picked struct
for simplicity at first and I don't really think it's a big deal. I know some C++ devs will probably object. This is more js style I guess 😄
@@ -39,17 +39,10 @@ | |||
"<(module_root_dir)/deps/leveldb/leveldb.gyp:leveldb" | |||
], | |||
"include_dirs" : [ | |||
"<!(node -e \"require('nan')\")" | |||
"<!(node -e \"require('napi-macros')\")" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
❤️ @mafintosh
|
||
function ChainedBatch (db) { | ||
AbstractChainedBatch.call(this, db) | ||
this.binding = db.binding.batch() | ||
this.context = binding.batch_init(db.context) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here's how we do it, regardless of where we are. We use an _init
method to get a context
.
@@ -43,7 +44,7 @@ | |||
"verify-travis-appveyor": "^3.0.0" | |||
}, | |||
"scripts": { | |||
"install": "prebuild-install || node-gyp rebuild", | |||
"install": "node-gyp-build", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
node-gyp-build
checks if the module can be require()
'd, if it fails it will recompile.
Picking some random people for review. It's fine if you don't have time or energy to review 😄 |
I opened #542 for further discussion after this PR; personally I don't agree. |
Wow, what a herculean effort! ❤️ |
There's a memory leak: running
Same with |
I'll have a look. Any tools we can use to track stuff down? This is one of my Achilles' heels. |
@vweevers Which version of node are you running? Just making sure we use the same setup. |
It was 8.6.0 x64 on Windows. I also tried 8.14.0, doesn't exhibit the same growth rate. I'm running 10.14.1 now, currently at 3 million Edit: still steady at 6 million |
Hmm this is strange. I'm on v10.14.1 and I'm running up to 1000% even on the master branch. |
Ai. |
This is more for consistency, there were variants on this. Also makes it more clear where the slices are created
@vweevers Question is now, do we squash this or make a merge commit? I'd personally like to merge and keep the rewrite history. |
Sure 👍 |
@ralphtheninja I think it's time for our perfect Neil Patrick Harris gif. I'll let you do the honors. |
@vweevers I'm curious how the napi stuff works on node 6, e.g. the build https://travis-ci.org/Level/leveldown/jobs/468365583 Has napi been backported to node 6? |
Yes, but I think it's still experimental, see https://nodejs.org/api/n-api.html#n_api_n_api_version_matrix |
Aaah ✔️ But apparently, it does work :) |
Why?
N-API
is the latest interface for native addons in node.js. It's a set of c functions that lie between your native addon and v8. Main benefit is that your addon is independent of the v8 version, which means it will be compatible with future versions of node.js.So, as long as your native code doesn't change, it doesn't need to be recompiled as new versions of node.js are released. This helps greatly with both maintenance and also when delivering code to the users of your module.
The downside is a loss of performance since you're no longer interfacing directly with v8.
Porting to N-API
There's been some work done already to port
leveldown
toN-API
but I decided to give it a go myself. Mainly because accepting a huge PR as a maintainer means a lot of work, it's not much more work to actually do it yourself.Also, personally, I've always felt that the v8 "lingo" and abstractions were difficult to understand. Basically, you need to understand how v8 works (to some extent) in order to write good native addons. Maybe it was just over my head.
Design
However, I think "workers" as a concept is easy to understand, so I've tried to keep the code similar to the
nan
approach in this sense.LevelDB
is a C++ library and the glue code was already written in C++. Also, I personally like virtual methods, which really come in handy when creating workers. You can abuse virtual methods as well, but if you keep it to a minimum (no multiple inheritance) it can be quite elegant.Another reason for this was for simplicity during rewrite. The main focus when migrating from
nan
toN-API
should be at the intersection of js and c++ land. The code in between should really not have to change that much.Correct. At first I wanted to keep a single file to make it easier during rewrite, but the more I thought of it, the more it made sense to keep everything in one amalgamated file. I'd prefer to keep it this way. It's so much easier to find things. Each async method has its worker right next to it.
Feedback
Since this PR is quite big, I'd prefer to keep review at a bigger level. Please let me know if there's something in the code that's a big no no, e.g. if I'm using the wrong abstractions.
Input such as "Did you know you can do X instead of Y and then you don't have to Z?" is also very helpful.
Another important thing is that when you rewrite things and there are many details to keep in your head, you often forget things. So if you can find something that I have forgotten, that'd be great.
One thing that would be greatly needed is also help with finding bugs. It would be really helpful if you can make the code segfault. Also running benchmarks to compare with the previous implementation would be nice.
Some stats
I tried running
test-for -n 100
for pure tests (e.g. only the tape tests, without standard etc) and got the same average for both themaster
and thenapi
branch (~3.8s/test). Timing tests is most likely not a good measure, but it's a indication at least.Total lines of native code in
src/
:Compared to
binding.cc
: